If we want to log in to Swestore through command line we have to use lftp.
If we are trying to use lftp through aurora lunarc cluster we have to login to our aurora account and then login to the Swestore.
There are two main ways to login to the aurora resources.
First, through SSH protocol as follows:
salma@salma:~$ ssh salka@aurora.lunarc.lu.se
Password:*************
Enter your Pocket Pass OTP:
The password is given to you by lunarc people on phone and Pocket Pass is available through Pocket Pass app on your phone (everytime you want to login to your aurora account!)
Second, through the graphical Thinklinc client interface as follows:
Now, it is the time to download files from Swestore through lftp.
It is essential to load any module we want to use in aurora before using and also all dependancies.
For finding all dependencies (e.g., lftp), following command works:
module spider lftp
And then it gives us some information on the packages and the vesrions that we have to load before our needed package:
module load GCCcore/6.4.6
module load lftp/4.8.4
lftp
lftp https://s_salka@webdav.swestore.se/snic/aits/
Password:
cd ok, cwd=/snic/aits
After accessing to Swestore files now we can use the following command for downloading and also uploading files, respectively:
get MFGTMPcx7_170615150001.tar.gz
put MFGTMPcx7_170615150001.tar.gz
Which works for multiple files either.
** For uing simple bash commands there should be a "!" before all commands in lftp environment.
A simple note on the bash scripts for job assignment on aurora¶
A simple bash script for a job assignment in aurora is as following:
#!/bin/bash
# Put in actual SNIC number
#SBATCH -A SNIC2019-6-17
# Asking for 1 node (the whole node) on Aurora
#SBATCH -n 1
#SBATCH -c 8
#SBATCH -J Aug26
#SBATCH --time=70:00:00
ml bftools
ml GCC/7.3.0-2.30 CUDA/9.2.88 OpenMPI/3.1.1
ml GCC/7.3.0-2.30 OpenMPI/3.1.1
ml icc/2018.3.222-GCC-7.3.0-2.30 OpenMPI/3.1.1
ml icc/2018.3.222-GCC-7.3.0-2.30 impi/2018.3.222
ml ifort/2018.3.222-GCC-7.3.0-2.30 OpenMPI/3.1.1
ml ifort/2018.3.222-GCC-7.3.0-2.30 impi/2018.3.222
ml Python/3.6.6
python3.6 convert_images.py > log_file.txt
The first lines are for the number of nodes we need, the snic number of the project and also the maximum time we need to run our job. If our job exceeds the time we metion in the script, the job will be stopped.
The next part is for loading the dependencies and also the packages are needed for our job.
The last step would be assigning the job by following commands and checking the reference number and the status of the run:
sbatch job.sh
squeue -u salka
For assigning python jobs on aurora, it is better that we build a virtual environment and add all packages locally rather than loading them. The reason is simply version conflict of different modules and their dependencies on aurora.
In order to do this we have to do following steps:
module purge
module load Anaconda3/5.2.0
Then we create the conda enviroment with:
conda create --prefix /lunarc/nobackup/users/salka/envs/python python=3.6.5 opencv scikit-learn glob2 (this is all on one line)
Proceed ([y]/n)? y
After creating the environment we'll have to activate it:
conda activate /lunarc/nobackup/users/salka/envs/python
Now we can install the needed packages: e.g.,
pip install Pillow==6.1.0
conda install -c conda-forge umap-learn Proceed ([y]/n)? y
pip install scipy==1.0.0
After we're done with running our script(s):
conda deactivate
Then we can simply run our bash script without all those ml lines!
We have a space of about 5TBs on nobackup drive of the aurora